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Abstract 


Adaptive automation (AA) has been explored as a solution to the problems associated with human- 
automation interaction in supervisory control environments. However, research has focused on the 
performance effects of dynamic control allocations of early stage sensory and information acquisition 
functions. The present research compares the effects of AA to the entire range of information processing 
stages of human operators, such as air traffic controllers. The results provide evidence that the 
effectiveness of AA is dependent on the stage of task performance (human-machine system information 
processing) that is flexibly automated. The results suggest that humans are better able to adapt to AA 
when applied to lower-level sensory and psychomotor functions, such as information acquisition and 
action implementation, as compared to AA applied to cognitive (analysis and decision-making) tasks. The 
results also provide support for the use of AA, as compared to completely manual control. These results 
are discussed in terms of implications for AA design for aviation. 
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Acronyms 


AA 

Adaptive Automation 

ATA 

Air Transport Association of America 

ATC 

Air Traffic Control 

DFA 

Dynamic Function Allocation 

SA 

Situation Awareness 

EDD 

Electronic Data Displays 

TPA 

Trajectory Projection Aids 

CDA 

Conflict Detection Aids 

CAA 

Clearance Advisory Aid 
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Electroencephalogram 

SD 

Standard Deviation 
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Introduction 


Automation refers to systems or methods in which many of the processes of production are 
automatically performed or controlled by autonomous machines or electronic devices” (Billings, 1997, p. 
7). Billings stated that automation is a tool, or resource, that allows the user to perform some task that 
would be difficult or impossible to do without the help of machines. Therefore, automation can be 
conceptualized as a process of substituting some device or machine for a human activity (Parsons, 1985). 
The dramatic increase in technology has significantly impacted all aspects of our daily lives. The 
Industrial Revolution ushered in an era of untold innovation that has not only made life easier and safer, 
but has also provided much more leisure time. One need only imagine washing one’s clothes on a 
washing board, something considered an innovation during the early 1900’s, to see how automation has 
transformed how we see ourselves and our place in the world. Automation has become so pervasive that 
many devices and machines are not even considered by most people to be “automated” anymore. Others, 
such as the modem airplane, however, do not escape visibility so easily. Wiener and Curry (1980), and 
Wiener (1989) noted that avionics has provided not only a dramatic increase in airline capacity and 
productivity coupled with a decrease in manual workload and fatigue, but also more precise handling, 
relief from certain routine operations, and more economical use of airplanes. Unlike the washing 
machine, the increased automation in airplanes and air navigational systems, however, has not developed 
without costs. 

The invention of the transistor in 1947 and the subsequent miniaturization of computer 
components have enabled widespread implementation of automation technology to almost all aspects of 
flight. The period since 1970 has witnessed an explosion in aviation automation technology. The result 
has been a significant decrease in the number of aviation incidents and accidents. However, there has 
also been an increase in the number of errors caused by human-automation interaction; in other words, 
those caused by “pilot error.” In 1989, the Air Transport Association of America (AT A) established a 
task force to examine the impact of automation on aviation safety. The conclusion was that, 

“During the 1970s and early 1980s.. .the concept of automating as much as possible was 
considered appropriate. The expected benefits were a reduction in pilot workload and 
increased safety... Although many of these benefits have been realized, serious questions 
have arisen and incidents/accidents have occurred which question the underlying 
assumption that the maximum available automation is ALWAYS appropriate or that we 
understand how to design automated systems so that they are frilly compatible with the 
capabilities and limitations of the humans in the system” (Billings, 1997 p. 4). 

A need exists to reconsider the development of advanced automated systems in aviation that truly support 
human-centered design. A recent approach has been termed, “adaptive automation.” However, although 
the concept of adaptive automation has been reported to have significant promise for mitigating 
“hazardous states of awareness” (Pope & Bogart, 1992) for flight crews (e.g., Haas, Nelson, Repperger, 
Bolia, & Zacharias, 2001; Prinzel, Freeman, Scerbo, Mikulka, & Pope, 2000; Prinzel, Pope, & Freeman, 
2002; Scallen & Hancock, 2001), adaptive automation has not received a similar amount of research 
focus in other aerospace domains, such as air traffic control (ATC). 

Automation for Aviation Systems 

In considering aviation systems, including air traffic control (ATC) workstations and aircraft, the 
current role of automation is restricted by the limitations of expert systems (Leroux, 1993). In general, 
automation is not capable of higher-order cognitive functions, such as information integration and 
decision-making, which are required for effective performance in ATC operations and piloting tasks 
(Leroux, 1993). Humans must remain part of decision-making processes in the control of such systems in 
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order to ensure optimal performance. The key limitation of automation for ATC is the lack of expert 
system capability to consider the context of a decision and to quickly select an alternative, as humans 
often do on the basis of decision making heuristics and biases. 

With these limitations in mind, some researchers (c.f., Laois & Giannacourou, 1995) have posed 
the question as to whether automation only should be applied to, for example, data acquisition and 
communication tasks versus it being applied to decision making functions or tasks requiring higher-order 
aspects of information processing. For example, Laois and Giannacourou (1995) stated that automation is 
generally better for monitoring tasks whereas humans are better at decision-making, especially in critical 
situations. That is, in the context of aviation system operations, automation is most suited to early sensory 
and information acquisition stages of infonnation processing while humans are well suited to the latter 
(i.e., advanced stages of processing). They studied human performance in an ATC simulation and 
surveyed expert controllers to determine the implications of automation of ATC decision-making 
functions on performance. They observed significant performance decrements when futuristic forms of 
automation (conflict projection and clearance advisory) were applied to decision functions in the 
simulation, particularly when high-level automation was used. The survey results indicated that 
automation only should be applied to data acquisition and communication versus conflict projection and 
clearance advisory. 

This past body of research suggests that caution should be exercised when considering the 
application of automation to ATC because of limitations in current technology and the implications of 
automation on human operator performance when applied to advanced functions, such as decision- 
making. The results found reported by Laois and Giannacourou (1995) demonstrate that automation of 
certain ATC functions may potentially undermine the overall objective of automation: To augment the 
Air Traffic Controller Operator skills. 

Brief Review of Contemporary Adaptive Automation Research 

Adaptive automation research has primarily focused on evaluation of performance and workload 
effects of dynamic allocations of control of early sensory and information acquisition functions as part of 
human-machine system operations. Kaber (1997), Kaber and Riley (1999) and Parasuraman et al. (2000) 
all reviewed a number of empirical studies of AA that have focused on the performance effects of 
Dynamic Function Allocation (DFA) in complex systems, specifically monitoring and psychomotor 
functions. On the basis of studies including Parasuraman (1993), Hilburn et al. (1993), Scallen et al. 
(1995) and Parasuraman et al. (1996), it is known that AA significantly improves monitoring and tracking 
task performance in multiple task scenarios, as compared to static automation and strictly manual control 
conditions. 

Unfortunately, little work has been conducted to establish the impact of AA on cognitive function 
performance (e.g., decision-making) or to make comparisons of human-machine system performance 
when AA is applied to various infonnation processing functions. The AA review literature has also 
pointed to the limited number of studies that have investigated the implications of DFA (a.k.a., adaptive 
automation) on cognitive task perfonnance. As one example, Hilbum et al. (1997) conducted a study of 
AA in the context of ATC to examine whether decision-making automation could be used to reduce 
operator workload and optimize performance. Specifically, they evaluated the use of an automated 
Descent Advisor that calculated aircraft trajectories and dynamically developed flight plans. The tool 
detected planning conflicts, or projected separation conflicts, and offered the human operator advice to 
resolve conflicts. Experienced Air Traffic Controllers were required to control an airport arrival traffic 
simulation with or without the assistance of the automation. Hilbum et al. (1997) used three automation 
schemes including constant manual control, constant automation and the AA condition (under which the 
automation was invoked only during high traffic conditions to simulate workload relief). They found that 
the AA condition resulted in the smallest increase in mental workload across trials. This research provides 
some support for the use of automation and/or AA in cognitive/decision making tasks. However, 
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additional research is needed to establish the relative effectiveness of AA applied to higher-order 
cognitive functions in comparison to AA of low-level sensory and information acquisition functions in 
specific contexts. This would provide additional insight into the utility of AA for addressing static 
automation problems across information processing functions. 

Some work that has indirectly investigated the implications of AA of lower-order aspects of 
human-machine system information processing has pointed to the need to study AA of the advanced 
stages of infomiation processing in complex systems, including decision-making and response execution. 
Crocoll and Coury (1990) evaluated the human performance consequences of automation reliability when 
applied to information acquisition and analysis as part of human-machine system performance. This work 
is relevant to the present research, as AA may be considered a form of unreliable automation. That is, 
depending upon the state of a system and its task, the automation may be turned “on” or “off’. This may 
or may not occur with operator notification. In the latter case, the operator may in fact perceive AA as 
unreliable automation. Crocoll and Coury ’s (1990) work attempted to define the conditions under which 
automation reliability does or does not affect human performance. They compared information 
acquisition/analysis automation with decision automation. Subjects that were provided information 
acquisition/analysis automation performed better than subjects that received decision automation or both 
forms of automation when the automation was unreliable. This research has shown that people can adapt 
to automation unreliability when computer control is applied to low-level information processing 
functions. It has also been suggested that negative effects of automation unreliability may be more 
pronounced for decision automation than for information analysis automation (Parasuraman et al., 2000). 
Parasuraman et al. (2000), the only study to look at this issue, pointed to the need to further examine 
whether automation unreliability has greater negative effects on the later stages of human-machine system 
information processing than on monitoring and infomiation analysis. 

Toward this end, Parasuraman et al. (2000) formulated a model-based approach to automation of 
complex systems (e.g., Air Traffic Control systems) based on existing theories of human information 
processing. Four stages of human-machine system information processing are considered in their model, 
including Information Acquisition, Infomiation Analysis, Decision-Making and Action Implementation. 
In addition, the level of automation of each stage is used to describe the overall degree of automation for 
the operation of a complex system. These stages correspond to aspects of human information processing 
included in historical pipeline models (e.g., Broadbent, 1958), such as perception, planning, decision- 
making, and action. The Parasuraman et al. (2000) model can be used to characterize various types of 
human-machine systems in terms of the aspects of infomiation processing required for effective 
performance. They may also serve to categorize the functions of human-machine systems in terms of 
operator information requirements and stages of infomiation processing. Therefore, the approach could be 
used to identify functions requiring higher-order cognition and facilitate examination of the application of 
AA to such functions and evaluation of the effect on human performance. In general, this method of 
automation design and evaluation needs to be evaluated through AA research. 

Objectives 

This research compared performance of a complex human-machine system under AA as applied 
to each of the four stages of human-machine system infomiation processing presented in the Parasuraman 
et al. (2000) model. The objective of this work was to establish the impact of AA on cognitive 
task/function perfonnance and to determine whether humans more easily adapt to dynamic allocations of 
psychomotor control functions than, for example, decision making. 

The project extended previous work that indirectly assessed the implications of A A on cognitive 
function performance. Kaber (1997) investigated the application of AA to a dynamic control task 
involving functions that represented general stages of infomiation processing, including formulating task 
processing plans and selecting among processing plan options. These functions required higher-order 
cognition of operators, including situation awareness (SA) and decision-making, for effective 
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performance. The functions were adaptively allocated between a human operator and computer based on 
predetennined allocation schedules (times when automation was turned “on” and “off’). Kaber (1997) 
found that low to intermediate degrees of system automation improved operator SA and perfonnance in 
comparison to manual control and full automation of all system frmctions. The adaptive allocation 
schedule appeared to significantly affect workload with longer periods of automation reducing subjective 
ratings of mental load. Unfortunately, the effects of AA for the planning and decision making functions of 
the dynamic control task were confounded by simultaneous AA of other functions, including system state 
monitoring and control action implementation. Therefore, the specific performance effect of AA of the 
decision-making component of the task could not be established. By studying AA of a complex system 
function representing a single stage of human information processing, while holding automation of all 
other frmctions fixed, the present research established the specific effect of AA on low-level sensory and 
psychomotor functions, as well as cognitive (unctions. 

It was hypothesized that humans would not be able to adapt to AA of decision making and 
information analysis frmctions of complex systems as well as they are able to use AA of information 
acquisition and psychomotor frmctions, including action implementation. It was further posited that 
application of AA to the decision making aspect of dynamic control task performance would not be as 
effective as AA applied to the monitoring or information acquisition aspects of the task for managing 
operator workload. 
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Method 


Participants 

Forty-seven North Carolina State University students were recruited for this experiment. 
Participants consisted of both graduates and undergraduates who ranged in age from 18 to 28, including 
both men and women. All participants possessed 20/20 or normal corrected vision and were nave to the 
task and its conditions. They were also required to have some degree of personal computer (PC) and 
video game experience. On a five-point scale ranging from one (“none”) to five (“frequently”), the mean 
for PC experience was 4.8, and the mean for video game experience was 3.5. Seven of the participants 
were used in a pilot study to establish criterion levels for various dependent measures recorded during the 
actual experiment. 

Tasks and Equipment 

Two computer-based tasks, a dynamic control task (Multitask) and a secondary gauge-monitoring 
task, were used in this experiment. Both of these tasks were modified versions of the tasks employed by 
Kaber and Endsley (1997) and Kaber and Riley (1999) in studies of the performance and workload effects 
of AA in dynamic control tasks and the effectiveness of a psychophysiological-based approach to AA 
imder different forms of DFA authority for managing operator workload. In the current experiment, the 
secondary task provided an index of primary task workload that was used to mandate automated control 
allocations. When operator performance in the secondary task fell below a predetermined level, the 
computer would mandate automated control of the primary task. Once performance in the secondary task 
returned to a level indicating an acceptable level of primary task workload, the Multitask simulation 
would return to manual control. There was no advance warning of the DFAs provided to operators. They 
were instructed to distribute their attention equally across both the secondary and the primary task. 

Both tasks were presented through high-resolution computer monitors at 1024 x 768 pixels. The 
gauge task was presented on a 17-inch monitor using an 850Mhz Pentium® III workstation and 
controlled by participants with a standard keyboard. Multitask was presented on a 21 -inch color monitor 
using an 800 MHz Pentium© laptop and controlled by participants with a 17-key numeric keypad and a 
mouse (see Figure 1). 
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Gauge Task 





Figure 1. Equipment layout. 


Secondary Task. The gauge-monitoring task presented a fixed-scale display with a moving 
pointer (see Figure 2). Subjects were required to monitor vertical pointer movements to detect when a 
deviation occurred from a central “acceptable” range on the scale (colored in “green”) into an 
“unacceptable” region (colored in “red”). The participants were required to correct for pointer deviations 
by pressing keys on the keyboard facilitating upward or downward pointer movements. Performance was 
recorded as a ratio of the number of off-nominal pointer deviations (i.e., the signal-to-noise ratio) missed/ 
number of total pointer deviations. 
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Figure 2. Secondary gauge-monitoring task display. 


Primary control task. The Multitask simulation presented subjects with a radar scope display and 
revealed the position of different types of aircraft in a simulated airspace through the scope (see Figure 3). 
The aircraft were graphically represented by three types of icons (military, commercial, and private), 
which moved at different speeds toward an airport, or home base, at the center of the display (see Figure 
4). The speed of each aircraft was dependent on its type. Military vehicles had the highest maximum 
speed, followed by commercial, and then private aircraft. All aircraft required between 60 and 120 
seconds to reach the center of the display after their initial appearance on the radar scope. 

The participant’s task was to locate and “clear” the aircraft for landing before they reached the 
center of the display or collided with another aircraft. Clearing an aircraft required two steps, including 
establishing a communication link and issuing a clearance. To establish a communication link, 
participants had to move a cursor to the location of an aircraft using the mouse, and then press the left 
mouse button. The aircraft icon then flashed for several seconds, signifying a processing stage. After the 
icon stopped flashing, the subject had to click on the aircraft again, but with the right mouse button in 
order to issue a clearance. The aircraft icon flashed again, but this time for a significantly longer period. 
Once the icon stopped flashing on the second occasion, the clearance had been issued and the aircraft 
could safely fly to the home base. Clearing each aircraft required at least 30 seconds (approximately 7 
seconds to establish a communications link and 23 seconds to process a clearance) and participants could 
clear multiple aircraft in parallel. 
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Figure 3. Primary control task display. 






HI 


Private 

Military 

Commercial 


Figure 4. Various aircraft types. 

During all training and experiment trials, the majority of the radar display was not visible to the 
participants. A small portion of the display was made visible through a portal, or “keyhole” (see Figure 
3), that could be moved by the participant in horizontal, vertical, and diagonal directions using the 
numeric keypad. Under certain Multitask simulation conditions, in order for subjects to clear aircraft from 
the radar scope, they had to first find them using the portal. The number and speed of the aircraft was set 
so it was nearly impossible to clear all vehicles appearing during each trial. In general, the simulation was 
designed to ensure that between five and six aircraft appeared on the radar scope at any given point in 
time. 

The version of Multitask used in this experiment provided one of five different modes of 
automated assistance to each participant. Each mode was designed to assist with a particular stage of task 
information processing as described below: 
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Information Processing Modes 

A Manual condition offered no assistance whatsoever. 

An automated Information Acquisition mode was designed to provide computer control of the 
movement of the portal, which followed an inward spiral toward the center of the display. By 
pressing a key on the numeric keypad, participants could optionally have the portal “lock-on” to 
aircraft as they were revealed through the automated movement of the keyhole. With this feature 
enabled, the portal would move in its regular pattern around the display until it revealed any part 
of an aircraft. Once an aircraft was located, the portal would center itself on the vehicle and 
continue to follow the aircraft’s path until the participant clicked with the right mouse button (to 
begin issuing a clearance) or released the portal from the aircraft by pressing another key on the 
keypad. This fonn of automation was considered an abstraction of implementing radar tracking 
systems into commercial ATC operations; that is, providing ATC with radar that actually tracks 
an aircraft versus using conventional scanning radar (Parasuraman et al., 2000). 

An automated Information Analysis mode presented a decision aid as part of the Multitask radar 
display, which showed a table of all aircraft currently on the radar scope along with their 
properties. These included the type of aircraft, its direction of travel, speed, distance from the 
center of the display (home base), stage of processing (communication link, clearance), and 
information on whether or not the aircraft might be involved in a collision. Information on each 
aircraft was presented in a random order. This type of automation was considered to be similar to 
futuristic fonns of ATC automation, including Electronic Data Displays (EDD), Trajectory 
Projection Aids (TPA), and Conflict Detection Aids (CDA) (c.f., Laois & Giannacourou, 1995). 

A Decision Making condition was designed to present a decision aid similar to that used in the 
information analysis mode, but without the speed, collision, and distance information. Instead, the 
decision-making decision aid sorted aircraft for subjects according to priority for processing, 
from the top to bottom of the table. Highest priority was given to aircraft on collision courses 
with other vehicles, then to those aircraft closest to the center of the scope. This fonn of 
automation was considered to resemble a Clearance Advisory Aid (CAA) in real ATC (c.f., Laois 
& Giannacourou, 1995), except subjects in this research were required to effect the instructions of 
the Multitask automation when they are provided. 

In the final mode, automated Action Implementation , a feedback display was integrated with the 
Multitask radar scope and presented the number of aircraft and their stage of processing. In this 
mode, participants only had to click on aircraft once in order to issue a clearance. The time to 
process aircraft was the same as in the other conditions, but the clearance was issued 
automatically after the communication link was established. Action implementation presented 
only five aircraft. 

With respect to measurement of subject performance in the Multitask simulation, the number of aircraft 
cleared by an operator was recorded and divided by the number of aircraft presented (during each minute 
of the simulation). This yielded a percentage of the total number of aircraft processed. During experiment 
trials, the various modes of automated assistance could be switched “on” or “off’ by the experimenter 
seamlessly through commands entered on a laptop, which presented an additional view of the primary 
task interface. Participants were unaware of the “Wizard of Oz” method, tenned used here for the way 
DFA was implemented, taking place. 
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Adaptive Automation of Primary Task Functions 

In general, this experiment was to study how the abstract manifestations of information 
acquisition and analysis, decision making and action automation impact human operator ability to 
function in complex system control. It was also conducted to establish whether dynamic control 
allocations of the various information processing (unctions between a human and computer could serve as 
an effective tool for managing human workload. The approach is akin to previous research (Laois & 
Giannacourou, 1995) that has attempted to identify forms of automated subsystems of ATC (e.g., EDD, 
TPA, CDA, CAA) that provide the greatest potential to aid human operators in their daily task 
completion. However, the present project focused on the adaptive delivery of automation and the potential 
implications of its use on cognitive task performance. 

There are a number of strategies to AA, or methods for triggering DFAs, which have been 
defined in the literature (see Scerbo (1996) for a thorough review) including: 

(A) Critical events - DFAs triggered by occurrence of events critically impacting system goals (e.g., 
malfunction) (Hilbum, Molloy, Wong & Parasuraman, 1993); 

(B) Performance measurement - DFAs triggered by degradations in human monitoring performance 
below a criterion measure (Parasuraman, 1993); 

(C) Psychophysical assessment - real-time assessment of operator workload (using for example 
physiological measures - electro-encephalogram (EEG) signals or heart-rate variability) as basis 
for decision to automate (Prinzel, Freeman, Scerbo, Mikulka, & Pope, 2000; Pope, Comstock, 
Bartolome, Bogart, & Burdette, 1994; Byrne & Parasuraman, 1996); and 

(D) Behavior modeling - DFAs occur to human and computer to achieve predetermined pattern of 
overall system functioning (Rouse, Geddes & Curry, 1986). 

(E) Similar to the psychophysical assessment strategy, Hancock and Chignell (1988) also proposed a 
strategy to AA involving comparison of current and future states of operator workload as well as 
system performance as a basis for DFAs. 

For this experiment, a workload-based approach was taken to adaptive allocation of the information 
processing functions as part of the Multitask simulation. Subject performance on the gauge-monitoring 
task (secondary task workload) detennined dynamic allocations of Multitask information processing 
functions to the human operator (manual control) or to the computer (automated control). Since the 
perceptual and cognitive demands of Multitask functions overlap those of the gauge-monitoring task, 
previous research (Kaber & Riley, 1999) has found the gauge-monitoring task to be a sensitive indicator 
of workload changes in the Multitask simulation, as effected by AA. 

During the training sessions as part of the experiment, the average gauge-monitoring performance 
level and the standard deviation (SD) for the hit-to-signal ratio on pointer deviations was recorded. 
During the experiment trials, when performance of the gauge-monitoring task dropped below 1 SD of 
average task performance recorded during training for a particular user, the user was shifted to automated 
control in the Multitask simulation. While under automated control, when perfonnance of the gauge- 
monitoring task reached 1 SD above average, users were returned to manual control of the Multitask 
simulation. These criteria were defined based on pilot data suggesting subject overload and underload at 
±1 SD about mean perfonnance, and Kaber and Riley’s (1999) use of the coefficient of variation for 
secondary task workload, as a basis for DFAs in a similar task scenario. 
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Experiment Design and Independent Variable 

A between-subjects design was used with modes, or levels, of automation as the independent 
variable in order to minimize the potential for Multitask training carry-over effects from one experimental 
trial to another. Four groups of subjects experienced AA as applied to one of the four stages of task 
information processing (information acquisition, information analysis, decision-making, action 
implementation). 

For comparison purposes, two control conditions were also studied as part of the experiment. A 
completely manual control group performed the Multitask simulation with no automated assistance. 
These subjects also performed the secondary task in order to ensure a fair comparison of overall human- 
machine system performance across the AA and completely manual control conditions. The second 
control condition involved full automation of all functions as part of Multitask operation. In this 
condition, the computer processed all aircraft automatically as they entered the simulated airspace 
(appeared on the radar display). As in all other conditions, each aircraft required 30 seconds for complete 
processing (7 seconds to establish a communication link and 23 seconds to issue a clearance); however, 
the search time required for human operators to locate an aircraft in the airspace was virtually eliminated. 
(The computer system generated aircraft for processing and, therefore, stored their locations during the 
simulation.) This condition was investigated to establish the maximum performance capability of the 
automation. No subjects were used in evaluation of this condition, as no human control was required. 
With respect to the four AA conditions and the completely manual control condition, each test subject 
performed two trials at his or her assigned level of automation (LOA). 

Response Measures 

As previously mentioned, performance in the Multitask simulation was primarily measured in 
terms of the number of aircraft cleared, divided by the number of vehicles presented. An additional 
performance measure was calculated based on the number of potential aircraft collisions divided by the 
number resolved by an operator. (A near-collision was recorded when the buffer zones surrounding two 
aircraft intersected.) It is important to note that the Multitask simulation was preset to simultaneously 
present five aircraft at any point in time under the information acquisition and action implementation 
modes of operation, and six aircraft under those AA conditions involving information analysis and 
decision making aiding. These settings were selected based on pilot tests revealing that subjects could 
acquire and clear all aircraft under the information analysis and decision making modes for lower 
numbers of aircraft and to ensure that the level of workload across all AA conditions was approximately 
comparable. In an attempt to balance workload across conditions, four pilot subjects subjectively rated 
mental workload in the dual-task scenario under various AA conditions using the Modified Cooper- 
Harper scale. Results revealed a general correspondence among ratings for the simulation settings 
identified above. 

As previously mentioned, perfonnance in the gauge task was measured as the hit-to-signal ratio 
on pointer deviations (the number of unacceptable pointer deviations divided by the total number of 
deviations presented). For both tasks, the computer systems recorded perfonnance observations on a per 
minute basis. The number of automation to manual control, or manual to automation mode, shifts that 
occurred during a trial was recorded along with the percentage of time spent in automated mode. 
Transitions between control modes only occurred at the end of a full minute during task perfonnance. The 
measures on mode shifts and time under automation were intended to capture any effect of AA applied to 
the various aspects of human-machine system information processing on the rate of changes in operator 
workload or the frequency of dynamic control allocations. Subjective workload assessments were also 
captured using the NASA-Task Load Index (TLX) scale. 
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Procedure 


The procedures for the experiment included: An introduction, completion of a background 
questionnaire and consent forms; 15 minutes of training in Multitask under the manual mode; 15 minutes 
of training in Multitask under the assigned LOA (subjects in the manual condition received a second 15 
minute manual training period); 5 minutes of training in the gauge task; 20 minutes of training in the 
dual-task scenario under the assigned Multitask LOA with 2 minute cycles of manual and automated 
control; two 20-minute trials under AA at the assigned LOA (or completely manual control for the 
subjects included in the control group); and completion of NASA-TLX demand component ratings after 
each test trial. 

A short break was provided to subjects after they completed the training trials and there was an 
extended (10 min.) break between the two test trials. During the first break between the final dual -task 
training session and the experimental trials, an experimenter calculated the average and SD of the hit-to- 
signal ratio for the gauge task performance. The first four minutes of the trial were excluded from this 
analysis, as this was the first time the subjects attempted the dual-task scenario and some time was 
allowed for them to become acquainted with the scenario. The mean and SD for each subject’s dual-task 
practice were then input into the gauge -monitoring application for the experimental trials as criteria for 
control mode shifts. 

During the experimental trials, an experimenter was notified when the performance criterion for a 
shift in control mode occurred. The notification also included a discreet beep from the computer 
presenting the gauge task. During all test trials, experimenters wore an earpiece in order to hear the sound 
from the gauge task without the subject being alerted as well. When a shift notification was given, an 
experimenter would press the space bar on the laptop computer that would change the mode of Multitask 
operation from manual control to automated control or vice-versa. This action was also done discreetly so 
that participants would not be aware that the experimenter was controlling the control mode shifts. 
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Data Analysis 


All dependent measures were subjected to a two-way Analysis of Variance (ANOVA) with Level 
of Automation (LOA) as a between-subjects variable and trial as a within-subjects variable. The 
experimental design was replicated in order to produce an error term for evaluating the impact of the 
various LOAs on human-machine system performance and operator workload, as well as the role of 
individual differences in the results. In an attempt to assess the effectiveness of the experimental training 
protocol and to determine whether trial carry-over effects may have occurred, the trial number was 
included in all initial statistical models. If the trial term did not prove to be significant in these initial 
analyses, it was removed from the statistical model and the ANOVA was re-run on a reduced model in 
LOA. Accordingly, if the trial term was significant, it was retained in the frill model. A LOA x trial 
interaction effect was also initially studied, but did not prove to be significant in the majority of the 
response models and was therefore dropped from those analyses. 

Duncan’s Multiple Range (MR) test was used to further investigate any significant effects 
revealed by the ANOVAs. An alpha-level of 0.05 was used to establish statistical significance of any 
effects and as a basis for identifying significantly different factor settings (except where noted otherwise). 
Residual and normal probability plots were generated in order to assess conformance of the experimental 
data with the assrmrptions of the ANOVA, including normality and constant variance. In addition, all data 
sets were subjected to Shapiro-Wilks test for normality. In the event that the Shapiro-Wilks test statistic 
was significant (p<0.05), appropriate transforms on the response measures and predictors were 
considered. If outliers were identified based on graphical analysis using the residual plots, a regression 
analysis was conducted on the data set using SAS and Cook’s D values, as well as the SAS DFFIT’s 
statistic, were determined in order to more objectively identify outlying observations. Based on the plots 
and the indicators of the strength of specific observation effects on the statistical model, outliers were 
identified. If both the Cook’s D value and the SAS DFFIT’s statistic indicated that a specific observation 
was an outlier, or if either one of the measures indicated that a specific observation was an extreme 
outlier, the observation was removed from the data set. 

Performance Measures 

For the statistical analyses of the performance measures, observations were separated into two 
sets of data, one that included perfonnance while in manual mode and a second that included perfonnance 
while in automated mode. For each subject. Multitask and gauge monitoring perfonnance was averaged 
across the automated minutes of a trial to obtain a single score for each trial. Similarly, perfonnance 
observations under manual minutes were averaged to obtain a single score for each trial. Thus, for each 
subject, there were four perfonnance measures, including: (1) Multitask perfonnance while under manual 
control; (2) Multitask perfonnance while under automated control; (3) gauge-monitoring performance 
while manually controlling the Multitask simulation; and (4) gauge-monitoring performance while 
Multitask was automated. 

With respect to performance under automation, only the four levels of the independent variable 
representing automation of the four stages of information processing were considered. Therefore, each 
data set included 64 observations (4 LOAs x 8 subjects x 2 trials). However, one subject logged 0 minutes 
under automated control in both of the test trials. Beyond this, using the method described above for 
identifying outliers, 5 observations on Multitask perfonnance under automated mode were removed from 
the data set. Thus, 57 observations remained for analysis. With respect to gauge-monitoring performance 
under automated control, 2 outliers were removed the data set, resulting in a total of 60 observations for 
analysis. 

In regard to the Multitask and gauge-monitoring performance measures under manual control, 
five levels of the LOA variable, including the completely manual control condition, were considered in 
the analysis. Consequently, each data set included 80 observations (5 LOAs x 8 subjects x 2 trials). With 
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respect to the data on Multitask performance under manual control, 3 outliers were removed, resulting in 
a total of 77 observations. Finally, two outliers were removed from the data on gauge-monitoring 
perfonnance under manual control, resulting in a total of 78 observations. 

Automation Shifts 

The total number of minutes under automated versus manual control was summed for each trial. 
This number was divided by 20 minutes (the test trial duration) to obtain the percent time under 
automated control for each trial. In addition, the niunber of shifts in the mode of control was counted for 
each trial. Both of these measures applied only to the four AA conditions; thus, the data sets included 64 
observations each (4 LOAs x 8 subjects x 2 trials). With respect to the number of minutes under 
automated control, 3 outliers were removed from the data set, resulting in a total of 61 observations. Only 
one outlier was removed from the data on the niunber of automation shifts during trials, resulting in a 
total of 63 observations for analysis purposes 

Workload Measures 

Following each trial, subjects rated task workload using the NASA-TLX. The individual demand 
component ratings were combined with the component rankings collected at the onset of the experiment 
in order to calculate an overall/ weigh ted workload score. This overall score and highly ranked demand 
components, including temporal load, were subjected to the two-way ANOVA described above. These 
analyses were completed both with and without observations on subjects included in the manual control 
condition. That is, the full data set of 80 observations was analyzed as well as the subset of 64 
observations that represented only the AA conditions. With respect to the overall workload scores, 2 
outliers were removed from the data; therefore, data sets including 78 and 62 observations were analyzed. 
In regard to the temporal load ratings, 4 outliers were removed the data. Consequently, data sets of 72 and 
60 observations were analyzed. 


Results 


Performance Measures 

Primary Task Performance. The analysis of Multitask perfonnance under automation revealed a 
significant effect due to trial (/■’( 1 ,32 )=9.26, /;<0.01 ) with perfonnance in the second trial being 
significantly superior. There was a trend in the data indicating an effect due to LOA, which proved to be 
marginally significant (F(3,27)=2.76, /><0 . 1 ). A post-hoc analysis of this trend using Duncan’s MR test 
with an alpha level of 0.10 revealed that performance under automation of action implementation may be 
substantially higher than perfonnance under the three other automation conditions (infonnation 
acquisition, information analysis, and decision making). Figure 5 presents the mean Multitask 
performance for each AA condition, frilly automated processing, and the completely manual control 
mode. The data on the fully automated condition was not included in the statistical analysis and is 
presented for comparison purposes only. Note: All figures present the information processing stages 
and/or trial data in the sequence and gray-scale shading as presented in the figure legends. 

The analysis of Multitask perfonnance under manual minutes indicated no significant main effect 
due to LOA (also shown in Figure 5). However, there was a significant LOA by trial interaction 
(/•’(4,32)=3.62, p<0.05). According to post hoc analysis, perfonnance in the manual mode of the primary 
task during the second trial for those subjects under the condition applying AA to the action 
implementation function was significantly better (p< 0.05) than manual perfonnance in all other LOAs 
and trials. In addition, the second trial of the manual control condition resulted in significantly poorer 
perfonnance than all of the other LOA conditions and trials, with the exception of the first trial of the 
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manual control condition and the first trial of the condition applying AA to the decision making aspect of 
the Multitask simulation (see Figure 6). Note: All graphs are presented in grayscale format and the 
shading of the graph bars correspond to legend index shading. Additionally, the legend index 
corresponds to the sequential graph bar presentation (e.g., first bar corresponds to “information 
acquisition”; second bar to “information analysis”). 
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Under Automation 


In Manual Mode 


Level of Automation 


Figure 5. Primary task performance under automated and manual control. 
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Multitask Performance During Manual Minutes 



Level of Automation 


Figure 6. Primary task performance during manual minutes by LOA and trial. 

Since the LOA by trial interaction generally indicated that there were greater differences due to LOA in 
the second trial than in the first, the data from the second trial was analyzed separately in order to 
determine whether there was a main effect due to LOA during the second trial. The analysis revealed a 
significant effect due to LOA (F(4,34)=2.98, p< 0.05). Duncan’s MR test revealed that perfonnance in the 
primary task under manual control as part of the condition applying AA to the action implementation 
function was significantly better than performance under manual control in the completely manual control 
condition (see Figure 7). 

It may have been possible that the second test trial was more sensitive for revealing performance 
differences due to the LOA as operators had the experience of the first test trial to refine their strategies to 
exploit the features of the AA as applied to the various information processing functions. It appeared that 
AA of action implementation positively influenced operator manual control of the simulation as part of 
this condition in comparison to strictly manual performance. In general. Figure 7 suggests that there was a 
trend for better manual control perfonnance as part of the AA conditions, as compared to the completely 
manual control condition. 
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Multitask Performance, Manual Minutes, Trial 2 
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Figure 7. Primary task performance while under manual control in the second trial. 

Secondary Task Performance. The analysis of performance on the gauge-monitoring task during 
automation of the Multitask simulation indicated a significant effect due to LOA (F( 3,27) = 3.41, 
p<0.05). Post hoc analysis revealed that the hit-to-signal ratio in the secondary task was significantly 
higher (indicating lower workload) when AA was applied to the information acquisition and action 
implementation functions of the primary task, as compared to when it was applied to information analysis 
(see Figure 8). The decision aids provided as part of the information analysis and decision-making 
conditions included additional visual displays (compared to automation of information acquisition and 
action implementation) that may have increased both visual attention and cognitive processing loads for 
operators leading to poorer gauge performance under automation of those conditions. During the manual 
minutes, there was no significant effect due to LOA (see Figure 8); however, there was a highly 
significant effect due to trial (F(l,40)=12.81, /><(). 005). Post-hoc analysis revealed gauge-monitoring 
performance (workload) to be significantly worse during the second trial than the first (see Figure 9). 
Although not significant, a similar difference in performance scores for the first and second trials was 
observed for automated control. 

It is possible that subjects may have been slightly more fatigued in the second trial, as compared 
to the first. Another possibility is that subjects shifted their attention away from the secondary task and to 
the primary task over time. This is supported by the increase in performance in the primary task during 
the second trial, as compared to the first; however, this observation was only statistically significant when 
automated control was used. 
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Figure 8. Gauge task performance by LOA imder automated and manual control. 
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Gauge Task Performance 



■ Trial 1 
□ Trial 2 


Figure 9. Gauge task performance by trial under automated and manual control. 

Automation Shifts 

The analysis of percent time under automation revealed no significant effect due to LOA, 
however, there was a significant main effect due to trial (/’( 1 ,32 )=9.44. p< 0.005) (see Figure 10). 
Duncan’s MR test revealed that the time spent in under automated control was significantly greater in the 
second trial than in the first. This result is related to the significant trial effect seen for gauge-monitoring 
performance. Since the shifts from manual control to automation in the primary task occurred when 
gauge-monitoring performance was poor, and performance in the gauge-monitoring task was generally 
worse during the second trial than in the first, it is logical that automation was invoked for longer periods 
in the second trial. 

Although there appeared to be a large difference in the percentage of time-on-task under action 
implementation automation (31%) versus information acquisition automation (18%), the difference was 
not statistically significant. Upon closer examination of the time data, it was observed that, on average, 
subject spent 26% of task time in an automated mode; however, the overall SD was 20%. 
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Figure 10. Average percent time under automation by LOA and by Trial. 

The analysis of frequency of shifts between control modes revealed a significant effect due to LOA 
(F(3,28) = 3.37, /><0.05) (see Figure 11). Post hoc analysis revealed significantly fewer automation 
control shifts under AA as applied to infonnation analysis compared to AA as applied to action 
implementation. 

The low number of shifts under the information analysis condition can be explained by referring 
to perfonnance in the gauge-monitoring task as part of this condition. Subjects performed worse under 
automation of the primary task than when using manual control (see Figure 8). However, the algorithm 
used to cause shifts in primary task control assumed perfonnance would improve under automation. 
Subjects in the infonnation analysis AA condition were less likely to shift to automated control since this 
required their manual control perfonnance during the test trials to be worse than their average 
performance under both automated and manual control modes recorded during practice. If they did shift 
into the automated control mode, they were even less likely to return to manual control since this required 
that their performance under automated control be better than their average perfonnance under both 
automated and manual control. 
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Frequency of Automation Shifts 



H Information Acquisition 
H Information Analysis 

□ Decision Making 

□ Action Implementation 


Figure 1 1 . Average number of shifts between automated and manual control by LOA. 

Workload Measures 

The analysis of the overall workload scores revealed no significant differences between the AA 
conditions and the completely manual control condition. The analysis of temporal load ratings revealed a 
trend due to LOA (F{434)=2.4, p<0.1).When the observations on the manual control condition were 
excluded from the analysis and only the AA conditions were compared, a significant effect due to LOA 
was revealed (F( 3,27)=3.03, p<0.()5). Post hoc analysis revealed that subjects assigned to the conditions 
applying AA to infonnation analysis and action implementation functions rated their temporal load higher 
than subjects experiencing AA applied to information acquisition (see Figure 12). When AA was applied 
to the information acquisition function, the search for aircraft on the radar scope was automated (the 
computer moved the portal) and the task, in general, became machine-paced instead of operator-paced. 
This may have caused subjects to perceive less time pressure in the completion of the task. 
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Temporal Load Rating 


■ Information Acquisition 

■ Information Analysis 

□ Decision Making 
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Figure 12. Average rating of temporal load across AA conditions. 




Discussion 


These results provide evidence that the effectiveness of AA is dependent on the stage of task 
performance (human-machine system information processing) that is flexibly, or adaptively, automated. 
In general, the final stage of action implementation appears to be well suited to AA. Primary task 
performance was greatest when automation was applied to the implementation aspect of the task. More 
importantly, manual control performance as part of AA of action implementation was better than under 
the other AA conditions and the completely manual control condition. This suggests that automation of 
the action implementation stage of processing influenced performance during the manual mode of 
operation. 

Considering the results on secondary task performance on the gauge task, or the workload 
measure, signal detection performance was greater for both automated and manual minutes when AA was 
applied to the information acquisition aspect of the task, particularly when compared to automated control 
as part of AA of the infonnation analysis function. The automation under the information acquisition 
condition appeared to relieve some task time pressure for subjects, as the computer automatically 
controlled the motion of the portal in searching for aircraft and allowed subjects more time to attend to the 
secondary task. In contrast, automation of infonnation analysis appeared to reduce the time available to 
attend to the secondary task. It is possible that the decision aid display as part of this condition held 
operator attention in their attempts to affect an optimal processing strategy. In general, the complexity of 
the automation and visual attention required of the displays may have caused an increase in primary task 
workload. 

Contrary to expectation, the results on control mode shifts demonstrated that the AA strategy 
investigated here was ineffective for managing operator workload under the information analysis. This 
was primarily due to the characteristics of the automation. The decision aid as part of this condition 
appeared to induce more cognitive processing of Multitask infonnation, specifically investigating 
potential collisions and prioritizing aircraft for clearances based on their characteristics. This may have 
lead to ineffective implementation of AA because the DFA trigger criterion used in the experiment 
assumed that automation would provide some workload relief to subjects. Beyond this, the AA strategy 
appeared to be highly sensitive to operator workload fluctuations when AA was applied to the 
psychomotor (action implementation) aspect of the task. It was initially hypothesized that the AA strategy 
would be less effective for the decision-making automation condition in comparison to the infonnation 
acquisition automation condition; however, there were no significant differences among these settings of 
LOA in terms of percentage of time-on-task under automation or the number of control mode shifts. 

All these results suggest that humans are better able to adapt to AA when applied to lower-level 
sensory and psychomotor functions, such as infonnation acquisition and action implementation, as 
compared to AA applied to cognitive (analysis and decision making) tasks. Finally, the results also 
provide support for the use of A A, as compared to completely manual control. In comparing performance 
under the manual control condition with both automated and manual control periods as part of the various 
AA conditions, performance was always better under AA with the exception of the first trial applying AA 
to the decision making function. This finding suggests that, given the fonns of AA investigated in this 
study, some AA, supporting early stage infonnation processing, is better than none at all. However, 
significant research questions still remain to be addressed including the effects of cognitive overhead, 
“trust” in automation and integrity, cognitive costs associated with different levels of IP support through 
adaptive automation, and how adaptive automation impacts situation awareness at different stages of 
information processing. Research is currently underway to address these and other human factors issues 
identified with this new, but exciting approach, to human-centered automation design. 
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